DEEP MODELS

DomiKnowS: A Library for Integration of Symbolic Domain Knowledge in Deep Learning (2021.emnlp-demo)

Challenge:	Existing protection schemes for deep neural network models protect intellectual property rights from being abused, stolen and plagiarized.
Approach:	They propose a practical approach for the IPR protection on recurrent neural networks without all the bells and whistles of existing IPR solutions.
Outcome:	The proposed approach is robust and effective against ambiguity and removal attacks on different RNN variants.

Copied to clipboard

Hossein Rajaby Faghihi, Quan Guo, Andrzej Uszok, Aliakbar Nafar, Parisa Kordjamshidi

Multimodality

Challenge:	Current deep learning architectures are data-hungry with issues mainly in generalizability and explainability.
Approach:	They propose a library for the integration of domain knowledge in deep learning architectures . structure of data is expressed symbolically via graph declarations and constraints can be added to deep models .
Outcome:	The proposed framework simplifies programming for integration of domain knowledge in deep learning architectures while separating the knowledge representation from learning algorithms.

Multiscale Collaborative Deep Models for Neural Machine Translation (2020.acl-main)

Copied to clipboard

Xiangpeng Wei, Heng Yu, Yue Hu, Yue Zhang, Rongxiang Weng, Weihua Luo

Shallow-to-Deep Training for Neural Machine Translation (2020.emnlp-main)

Challenge:	Neural machine translation models with deeper neural networks are difficult to train.
Approach:	They propose a MultiScale Collaborative framework to boost gradient back-propagation . they let each encoder block learn a fine-grained representation and enhance it .
Outcome:	The proposed framework outperforms baseline models on translation tasks with three translation directions and achieves a BLEU score of 30.56 on the English-to-German task.

Copied to clipboard

Bei Li, Ziyang Wang, Hui Liu, Yufan Jiang, Quan Du, Tong Xiao, Huizhen Wang, Jingbo Zhu

An Empirical Comparison of Instance Attribution Methods for NLP (2021.naacl-main)

Challenge:	Experimental results show that deep training is 1:4 faster than training from scratch.
Approach:	They propose a shallow-to-deep training method that learns deep models by stacking shallow models.
Outcome:	The proposed method is 1:4 faster than training from scratch and achieves BLEU scores of 30:33 and 43:29 on two translation tasks.

Copied to clipboard

Pouya Pezeshkpour, Sarthak Jain, Byron Wallace, Sameer Singh

Information Retrieval

Challenge:	Influence functions provide machinery for identifying training instances that may have led to a specific prediction, but are computationally expensive and prohibitive in many cases.
Approach:	They evaluate the degree to which different potential instance attribution agrees with respect to the importance of training samples.
Outcome:	The proposed methods exhibit desirable characteristics similar to more complex methods, but are computationally expensive.

White-box Testing of NLP models with Mask Neuron Coverage (2022.findings-naacl)

Copied to clipboard

Arshdeep Sekhon, Yangfeng Ji, Matthew Dwyer, Yanjun Qi

bert2BERT: Towards Reusable Pretrained Language Models (2022.acl-long)

Challenge:	Recent research has shown that black-box testing is not applicable to NLP models.
Approach:	They propose a set of white-box testing methods that are customized for transformer-based NLP models and adapt them to a black-box test suite.
Outcome:	The proposed methods can reduce testing suites by 60% while retaining failing tests, thereby concentrating faultdetection power of the test suite.

Copied to clipboard

Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, Qun Liu

On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Study (2023.findings-emnlp)

Challenge:	Pre-training large language models can be expensive and wasteful.
Approach:	They propose a method which can transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and a two-stage learning method to further accelerate the pre-training.
Outcome:	The proposed method can transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and significantly improve the pre-training efficiency of the large model.

Copied to clipboard

Polina Zablotskaia, Du Phan, Joshua Maynez, Shashi Narayan, Jie Ren, Jeremiah Liu

Text Generation

Challenge:	Modern deep models for summarization generate miscalibrated predictive uncertainty, compromising reliability and trustworthiness in real-world applications.
Approach:	They propose to use probabilistic methods to improve the uncertainty quality of neural summarization models by using three large-scale benchmarks with varying difficulty.
Outcome:	The proposed methods consistently improve the model’s generation and uncertainty quality, leading to improved selective generation performance (i.e., abstaining from low-quality summaries) in practice.

UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression (2022.emnlp-main)

Copied to clipboard

Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, Xiaodan Liang

Reasoning

Challenge:	Existing work on geometry problem solving treats calculation and proving as two specific tasks hindering a deep model to unify reasoning ability on multiple math tasks.
Approach:	They propose a large-scale Unified Geometry problem benchmark to unify geometry on multiple math tasks.
Outcome:	The proposed framework outperforms the existing model with 5.6% and 3.2% accuracies on calculation and proving problems.

Multi-layer Representation Fusion for Neural Machine Translation (C18-1)

Copied to clipboard

Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, Jingbo Zhu

Spec-o3: A Tool-Augmented Vision-Language Agent for Rare Celestial Object Candidate Vetting via Automated Spectral Inspection (2026.acl-long)

Challenge:	Neural machine translation systems require a number of stacked layers for deep models, but the prediction depends on the sentence representation of the top-most layer with no access to low-level representations.
Approach:	They propose a multi-layer representation fusion approach to fusing stacked layers to learn a better representation from the stack.
Outcome:	The proposed approach yields 0.92 and 0.56 BLEU points over the strong Transformer baseline on IWSLT German-English and NIST Chinese-English MT tasks respectively.

Copied to clipboard

Minghui Jia, Qichao Zhang, Ali Luo, Linjing Li, Shuo Ye, Hailing Lu, Wen Hou, Dongbin Zhao

Multimodality

Challenge:	Spec-o3 is a tool-augmented vision-language agent that performs astronomer-aligned spectral inspection.
Approach:	They propose a tool-augmented vision-language agent that performs astronomer-aligned spectral inspection via interleaved multimodal chain-of-thought reasoning.
Outcome:	Spec-o3 outperforms traditional visual inspection methods on rare-object inspection tasks.

Mixture of Attention Heads: Selecting Attention Heads Per Token (2022.emnlp-main)

Copied to clipboard

Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, Zhang Xiong

On Length Divergence Bias in Textual Matching Models (2022.findings-acl)

Challenge:	Mixture-of-Experts (MoE) networks have been proposed as an efficient way to scale up model capacity and implement conditional computing.
Approach:	They propose a new architecture that combines multi-head attention with the MoE mechanism and a sparsely gated architecture that allows for faster computations.
Outcome:	The proposed architecture can scale up the number of attention heads and the number parameters while preserving computational efficiency.

Copied to clipboard

Lan Jiang, Tianshu Lyu, Yankai Lin, Meng Chong, Xiaoyong Lyu, Dawei Yin

Topic Spotting using Hierarchical Networks with Self Attention (N19-1)

Challenge:	Existing deep models have been successful in textual matching tasks, but it is unclear whether they understand language or measure semantic similarity of texts.
Approach:	They propose an adversarial evaluation scheme which invalidates the length divergence bias in TM datasets.
Outcome:	The proposed method improves the robustness and generalization ability of models at the same time.

Copied to clipboard

Pooja Chitkara, Ashutosh Modi, Pravalika Avvaru, Sepehr Janghorbani, Mubbasir Kapadia

Dialogue Systems & Conversational Agents

Challenge:	Existing systems struggle to have consistent long term conversations with the users and fail to build rapport.
Approach:	They propose a hierarchical model with self attention for topic spotting . they compare it to previous proposed techniques for topic detection .
Outcome:	The proposed model outperforms existing models for topic spotting and deep models for text classification in an online setting.

R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling (2021.acl-long)

Copied to clipboard

Xiang Hu, Haitao Mi, Zujie Wen, Yafang Wang, Yi Su, Jing Zheng, Gerard de Melo

A Study of Reinforcement Learning for Neural Machine Translation (D18-1)

Challenge:	Existing models with stacked layers do not explicitly model hierarchical structure of language understanding.
Approach:	They propose a recursive Transformer model based on differentiable CKY style binary trees to emulate hierarchical composition process.
Outcome:	The proposed model can predict words given their left and right abstraction nodes.

Copied to clipboard

Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, Tie-Yan Liu

Saliency Learning: Teaching the Model Where to Pay Attention (N19-1)

Challenge:	Recent studies have shown that reinforcement learning (RL) is an effective approach for improving the performance of neural machine translation systems.
Approach:	They propose to leverage reinforcement learning to boost the performance of NMT systems trained with monolingual data.
Outcome:	The proposed method achieves competitive results on translation tasks in English-German, Chinese-English and English-English systems.

Copied to clipboard

Reza Ghaeini, Xiaoli Fern, Hamed Shahbazi, Prasad Tadepalli

Explainability & Interpretability in NLP

Challenge:	Recent work on explanation and interpretation has introduced methods to provide insights toward the model’s behaviour and predictions, but they do not improve the model's reliability.
Approach:	They propose to provide explanation training and ensure alignment of model’s explanation with ground truth explanation to ensure the model makes correct predictions for the right reason.
Outcome:	The proposed method produces more reliable predictions while delivering better results compared to traditional models.

ERASER: A Benchmark to Evaluate Rationalized NLP Models (2020.acl-main)

Copied to clipboard

Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, Byron C. Wallace

Chain of Thought Prompting Elicits Knowledge Augmentation (2023.findings-acl)

Challenge:	State-of-the-art models in NLP are opaque in terms of how they come to make predictions.
Approach:	They propose to release a benchmark to measure the quality of rationales extracted by models and how faithful these rationale are to human annotators.
Outcome:	The proposed benchmark will enable researchers to compare models and track progress on interpretable models for NLP.

Copied to clipboard

Dingjun Wu, Jing Zhang, Xinmei Huang

Semantic Text Processing

Challenge:	Existing knowledge augmentation methods require retrieving knowledge from external knowledge sources or developing a reasoner to leverage the logical rules within the external knowledge source.
Approach:	They propose a Chain-of-Thought-based method that augments knowledge for deep learning by removing the need for additional knowledge retrieval or knowledge reasoning models.
Outcome:	The proposed method outperforms both pure CoT-based methods and the non-augmented method across the majority of 11 publicly available benchmarks for various reasoning tasks.

Sampling Bias in Deep Active Classification: An Empirical Study (D19-1)

Copied to clipboard

Ameya Prabhu, Charles Dognin, Maneesh Singh

A Structural Probe for Finding Syntax in Word Representations (N19-1)

Challenge:	Existing studies on active learning identify sampling bias in large datasets . cost and time needed for labeling and model training are bottlenecks preventing new and/or better models from being trained .
Approach:	They propose to use active learning to identify representative data samples for training . they propose to create tiny datasets that can be used for cheap training if needed .
Outcome:	The proposed model outperforms the state-of-the-art on active text classification using small representative datasets with active learning.

Copied to clipboard

John Hewitt, Christopher D. Manning

Representation Learning

Challenge:	Existing methods for detecting syntactic knowledge do not test whether syntax trees are embedded in a linear transformation of a neural network’s word representation space.
Approach:	They propose a structural probe which evaluates whether syntax trees are embedded in a linear transformation of a neural network’s word representation space.
Outcome:	The proposed model shows that entire syntax trees are embedded in deep models’ vector geometry.

Copied to clipboard

John Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, James Glass

Representation Learning

Challenge:	Existing and novel similarity measures are used to analyze contextual word representations . different architectures have rather similar representations, but different individual neurons.
Approach:	They propose a method to analyze contextual word representation models using similarity analysis.
Outcome:	The proposed approach can be used to analyze model similarity without external annotations.

Revisiting Character-Based Neural Machine Translation with Capacity and Compression (D18-1)

Copied to clipboard

Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, Wolfgang Macherey

Simple Recurrent Units for Highly Parallelizable Recurrence (D18-1)

Challenge:	Translating characters instead of words or word-fragments can simplify the processing pipeline but results in longer sequences .
Approach:	They propose to use sequence-to-sequence architectures of sufficient depth to solve the problem . they also evaluate the performance versus computation time tradeoffs they offer .
Outcome:	The proposed models outperform models operating over word fragments in character-level NMT, the authors show . they also show that the proposed models do not match the performance of their deep character baseline model .

Copied to clipboard

Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, Yoav Artzi

Relating Simple Sentence Representations in Deep Neural Networks and the Brain (P19-1)

Challenge:	recurrent neural networks scale poorly due to the intrinsic difficulty in parallelizing their state computations.
Approach:	They propose a simple recurrent unit that provides expressive recurrence and allows highly parallel implementation.
Outcome:	The proposed model achieves 5—9x speed-up over cuDNN-optimized LSTM on classification and question answering datasets and delivers stronger results than LS and convolutional models.

Copied to clipboard

Sharmistha Jat, Hao Tang, Partha Talukdar, Tom Mitchell

Semantic Text Processing

Challenge:	Existing deep learning models for natural language processing are not fully studied.
Approach:	They investigate whether deep recurrent models learn sentences against those encoded by the brain and whether there is any correspondence between hidden layers of these models and brain regions when processing sentences.
Outcome:	The proposed models can be used to synthesize brain data and improve subsequent stimuli decoding accuracy.

Prototypical Extreme Multi-label Classification with a Dynamic Margin Loss (2025.naacl-long)

Copied to clipboard

Kunal Dahiya, Diego Ortego, David Jimenez-Cabello

Semantic Text Processing

Challenge:	Recent work in XMC addresses this problem using deep encoders that project text descriptions to an embedding space suitable for recovering the closest labels.
Approach:	They propose a method that uses a shallow transformer encoder to combine text-based embeddings, label centroids and learnable free vectors to improve XMC efficiency.
Outcome:	The proposed method achieves state-of-the-art in several public benchmarks of different sizes and domains while keeping the model efficient.

Domain-Specific Sentiment Lexicons Induced from Labeled Documents (2020.coling-main)

Copied to clipboard

SM Mazharul Islam, Xin Dong, Gerard de Melo

Sentiment Analysis

Challenge:	Existing sentiment lexicons reflect abstract notion of polarity and do not do justice to substantial differences of word polarities between domains.
Approach:	They propose to use domain-specific sentiment lexicons to induce initial word intensity scores and train new deep models based on word vector representations to overcome the scarcity of the seed data.
Outcome:	The proposed models show that they perform well on review classification and cross-lingual word sentiment prediction.

On the Calibration of Large Language Models and Alignment (2023.findings-emnlp)

Copied to clipboard

Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, Zhendong Mao